Folder Structure

Directory Tree #

M3ED/
├── input/
│   ├── raw_bags/
│   │   ├── car_urban_camera_calib_1.bag
│   │   ├── tower_imu_calib_1.bag
│   │   ├── car_urban_day_city_hall.bag
│   │   └── ...
│   └── lidar_calibrations/
│       ├── car_urban_day_city_hall.npz
│       └── ...
└── processed/
    ├── car_urban_day_city_hall/
    │   ├── car_urban_day_city_hall.h5
    │   ├── events_gray.{avi, mp4}
    │   ├── rgb.{avi, mp4}
    │   ├── car_forest_data_1_stats.yaml
    │   ├── car_urban_day_city_hall_pose_gt.h5
    │   ├── car_urban_day_city_hall_depth_gt.h5
    │   ├── depth.{avi, mp4}
    │   ├── depth_events.{avi, mp4}
    │   ├── car_urban_day_city_hall_semantics.h5
    │   ├── car_urban_day_city_hall_semantics.{avi, mp4}
    │   ├── car_urban_day_city_hall.pcd
    │   └── local_scans/
    ├── car_urban_camera_calib_1/
    │   ├── car_urban_camera_calib_1.h5
    │   ├── camchain.yaml
    │   ├── report-cam.pdf
    │   └── results-cam.txt
    ├── tower_imu_calib_1/
    │   ├── imu_chain.yaml
    │   ├── imu_report.pdf
    │   └── imu_results.txt
    └── ...

Input Files #

These are the files that are used by the build system to generate the h5 files.

• raw_bags: stores the unprocessed data obtained from the sensor tower. These files are not time corrected and events are stored in binary format.
• lidar_calibrations (npz): stores the manually tuned LiDAR calibrations. Multiple bags can share the same calibration file.

Processed Files #

Data Files (Train) #

Each sequence is recorded individually in the processed folder.

• data.h5: time synchronized and event decoded. Use these files to access the event camera data, the grayscale and RGB imagers, the lidar, and the IMU.
• events_gray.{avi, mp4}: recording of the event cameras and the grayscale imagers.
• depth_gt.h5: ground truth depth and poses from FasterLIO, for the left event camera. Use these files to access ground truth depth.
• pose_gt.h5: similar to depth_gt.h5, but without the depth information to make this bag lighter.
• depth_gt.{avi, mp4}: recording of the depth on the frame of the left event camera.
• depth_gt_events.{avi, mp4}: recording of the GT depth on the frame of the left event camera with events overlaid.
• stats.yaml statistics for the file.
• semantics.h5 InternImage semantic outputs.
• semantics.{avi, mp4}: recording of semantics on the frame of the left event camera with events overlaid.
• global.pcd: integrated point cloud for the whole sequence.
• local_scans: folder with individual scans from FasterLIO.

For videos, the extension defines the encoder used:

• avi: Uncompressed (FFV1) video. Allows to see details and noise that may be lost in compressed videos.
• mp4: Compressed with H264. Use these files for quick visualization.

Data Files (Test) #

Test files only provide:

• data.h5: time synchronized and event decoded. Contains only the events, grayscale, and the IMU. All the other sensors have been stripped.
• events_gray.{avi, mp4}: recording of the event cameras and the grayscale imagers.

Calibration Files #

Camera Calibration #

Camera calibration is performed with Kalibr multiple camera calibration. For each one of the recordings, the following results are provided:

• data.h5: time synchronized and event decoded.
• Kalibr outputs:
  • camchain.yaml: resulting camera chain with all the transformations in yaml format.
  • results-cam.txt: results summary as a text file.
  • report-cam.pdf: report in pdf format with all the plots.

IMU Calibration Files #

Camera-to-IMU calibration is performed with Kalibr camera-IMU calibration. For each one of the recordings, the following results are provided.

Please note that the IMU to camera calibration is only performed using the grayscale and RGB sensors from the OVC, and not the event cameras.

• data.h5: time synchronized and event decoded.
• Kalibr outputs:
  • imu-chain.yaml: resulting camera chain with IMU to camera transformations
  • imu_results.txt: results summary as a text file.
  • imu_report.pdf: report in pdf format with all the plots.